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Abstract 

In  robotics  and  other  manufacturing  applications,  real-time  supervisory  control  is  often  re- 
quired to  coordinate  several  different  sensors  and  actuators,  as  well  as  to  communicate  with  other 
computers  which  perform  higher-level  tasks  such  as  motion  planning  or  factory-level  coordina- 
tion. In  this  paper  we  describe  SAGE,  an  operating  system  designed  for  real-time  supervisory 
control  applications. 

The  SAGE  kernel  runs  on  a  Motorola  68020  processor  board,  and  provides  a  variety  of 
services,  including  process  and  memory  management,  protocol  support,  and  precise  timing  fa- 
cilities. In  addition,  filesystem  capabilities  are  provided  through  remote  procedure  calls  made 
over  a  LAN  interface.  Development  tools,  which  run  on  a  SUN  workstation,  allow  programs  to 
be  compiled  and  dynamically  downloaded  to  the  SAGE  host  as  needed. 

1      Introduction 

Most  robotics  operating  systems  have  concentrated  on  supporting  servo-level  control  tasks  [1,2,3,4] 
[5,6,7].  Such  tasks  are  typically  high  duty  cycle  control  loops  that  are  run  at  periodic  intervals.  The 
systems  built  to  support  these  control  applications  are  generally  minimal  executives  that  emphasize 
shared  memory  and  mailbox  techniques. 

Although  such  servo-level  control  systems  are  important  for  robotics  control,  in  an  integrated 
manufacturing  environment  there  is  also  a  need  for  higher  level  supervisory  control,  which  can 
perform  such  tasks  as: 

•  coordinate  multiple  sensors,  actuators,  and  sub-systems  to  form  a  coupled  system. 

•  communicate  with  other  computers  to  perform  higher-level  tasks  such  as  motion  planning  or 
factory-level  coordination. 

•  provide  support  functions  such  as  task  monitoring  and  error  recovery. 

Indeed,  most  of  the  proposed  robotics  control  architectures  either  explicitly  [1,3,4]  or  implicitly  [2,5,6] 
assume  the  presence  of  a  supervisory  control  system,  whose  job  is  to  interface  to  both  servo-level 
tasks  as  well  as  other  systems  in  a  distributed  hierarchical  fashion. 

Supervisory  control  systems  differ  from  servo-level  systems  m  several  ways.  First,  because  much 
of  the  servo-level  processing  has  been  offloaded,  supervisory  tasks  have  less  stringent  real-time  con- 
straints than  servo-level  tasks  (typically,  supervisory  tasks  have  "soft"  deadlines  on  the  order  of  mil- 
liseconds, while  servo-level  tasks  have  "hard"  deadlines  occurring  with  sub-millisecond  frequency). 
Secondly,  since  supervisory  control  systems  are  often  driven  by  external  events  happening  at  unpre- 
dictable times,  the  system  must  support  both  periodic  and  sporadically  generated  tasks.  In  contrast, 
most  servo  and  real-time  systems  are  designed  to  support  only  periodically  generated  tasks.  Finally, 
since  one  of  the  system's  main  requirements  is  to  communicate  with  other  machines  on  the  fac- 
tory floor,  supervisory  systems  place  a  much  greater  emphasis  on  distributed  communication  needs 
than  servo-level  systems.  The  net  effect  is  that  supervisory  systems  generally  must  provide  greater 
functionality  than  servo  systems,  while  still  insuring  real-time  response  on  the  order  of  a  millisecond. 
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Figure  1;  SAGE  architecture 

So  far,  however,  the  robotics  community  has  paid  little  attention  to  developing  supervisory  control 
systems.  In  general,  any  needed  supervisory  control  is  provided  by  a  general  purpose  computer 
system  [1,2,8],  a  commercially  available  real-time  system  [9,4],  or  directly  integrated  into  the  servo 
level  control. 

Unfortunately,  because  these  systems  have  been  designed  for  other  tasks,  most  of  them  are  not 
well  suited  for  supervisory  control  applications.  General  purpose  systems  often  do  not  provide  the 
real-time  response  required,  while  existing  real-time  systems  often  are  missing  needed  functionality 
[10].  Therefore,  if  the  performance  and  functionality  of  the  entire  control  system  is  to  improve, 
special  purpose  supervisory  control  systems,  in  addition  to  other  systems,  will  be  needed. 

SAGE  is  an  operating  system  designed  specifically  for  developing  and  supporting  real-time  su- 
pervisory control  applications.  The  system  provides  a  number  of  standard  interfaces,  protocols, 
and  support  libraries,  in  order  to  reduce  the  total  effort  required  to  interface  a  new  device  into  the 
system.  Several  debugging  tools  also  exist  to  track  down  and  isolate  errors  quickly. 

In  many  respects,  SAGE  is  similar  to  more  conventional  operating  systems,  since  it  provides 
multi-tasking,  memory  management,  communications,  and  a  number  of  supported  devices.  What 
is  unusual  about  SAGE  is  that  all  these  facilities  are  provided  with  relatively  low  system  overhead 
and  process  scheduling  latency. 


2      System  Architecture 

A  simple  architecture  was  adopted  for  SAGE,  as  shown  in  Figure  1.  SAGE  programs  are  developed 
and  cross-compiled  on  a  SUN  UNIX  workstation,  and  then  dynamically  downloaded  (usually  over 
an  Ethernet  or  other  LAN)  to  the  SAGE  host,  which  consists  of  a  resident  kernel  running  on  a 
Motorola  68000  family  processor  board.  Download  requests  can  be  initialed  either  remotely  from  a 
host  on  the  Ethernet  or  directly  from  the  SAGE  host. 

Although  the  system  can  support  a  number  of  servo-level  tasks,  a  SAGE  application  typically 
will  off-load  the  servo-level  tasks  to  a  number  of  dedicated  processors.  These  servo-level  tasks  can 
then  communicate  with  SAGE  processes  through  a  variety  of  device  interfaces  and  communication 
protocols.  SAGE  processes  avoid  polling  by  having  the  servo-level  tasks  either  signal  or  interrupt 
upon  completion  and  exceptional  conditions. 


SAGE  processes  invoke  system  services  through  traps  that  execute  in  the  kernel's  protected 
address  space.  Only  time  critical  or  privileged  operations  are  generally  implemented  in  the  SAGE 
kernel:  multi-tasking,  scheduling  and  process  control,  precise  timing  facilities,  memory  management, 
network  communications,  and  support  for  a  number  of  devices.  Other  generally  useful  facilities,  such 
as  file  access,  are  implemented  sis  remote  procedure  calls  made  to  a  server  process  running  on  a  UNIX 
host.  By  relegating  such  functions  to  the  SUN  host,  the  SAGE  kernel  and  support  software  can  be 
kept  relatively  simple. 

SAGE  system  primitives  are  fairly  low  level,  in  order  that  a  smart  application  may  handcraft 
its  own  environment.  More  extensive  system  facilities  are  provided  by  library  routines  which  invoke 
the  underlying  primitives.  For  instance,  many  of  the  UNIX  system  calls  are  available  to  SAGE 
programmers,  either  directly  or  through  library  emulation  routines. 

Currently,  SAGE  supports  programs  written  in  C,  C++,  or  Fortran.  The  system  uses  the  SUN 
compilers  and  loader  for  generating  object  code,  which  can  then  be  downloaded  to  the  SAGE  host. 
The  SAGE  kernel,  as  well  as  most  of  the  support  libraries,  is  written  in  C.  The  system  supports  most 
of  the  standard  UNIX  C  library  routines,  including  standard  I/O  and  dynamic  memory  allocation. 
Floating  point  is  fully  supported,  either  through  software  or  by  the  68881  co-processor. 

SAGE  encourages  the  building  of  hierarchical  control  systems  by  supporting  a  number  of  com- 
munication disciplines  and  interfaces.  In  particular,  the  system  provides  protocol  support  for  com- 
municating over  a  local  area  network,  allows  device  registers  to  be  mapped  into  a  process's  address 
space,  and  supports  a  number  of  standard  serial  and  parallel  interfaces. 

3      SAGE  Kernel 

We  overview  some  of  the  major  components  of  the  SAGE  kernel,  and  describe  why  these  features 
were  necessary  in  our  supervisory  control  applications. 

3.1  Multitasking 

Because  a  supervisory  control  system  must  typically  control  a  number  of  devices  and  communication 
streams  simultaneously,  each  with  relatively  low  (but  unpredictable)  duty  cycle,  SAGE  was  designed 
as  a  multitasking  system.  This  allows  an  application  to  dedicate  a  task  (or  several  tasks)  to  each 
external  device  or  stream. 

There  are  two  types  of  tasks  in  SAGE: 

•  (full-weight)  processes,  each  of  which  has  its  own  stack  and  memory  management  context,  and 
cam  invoke  the  full  range  of  system  services.  In  particular,  a  process  can  be  blocked,  suspended, 
or  preempted. 

•  interrupt  handlers,  each  of  which  runs  on  the  current  process's  kernel  stack,  can  access  any 
valid  kernel  address,  and  can  only  invoke  a  limited  number  of  operations.  Because  it  shares  the 
process's  stack,  an  interrupt  handler  cannot  block,  and  it  can  only  be  preempted  by  a  higher 
priority  interrupt  handler. 

In  general,  full-weight  processes  are  appropriate  for  supervisory  control,  because  the  event  driven 
nature  of  the  applications  imply  that  a  process  may  be  suspended  and  resumed  at  unpredictable 
times.  The  kernel  is  structured  so  that  full-weight  processes  can  be  preempted  at  most  places  inside 
the  kernel,  which  allows  worst  case  context  switch  latency  to  be  bounded.  Interrupt  handlers  are  used 
mostly  to  awaken  tasks  sleeping  on  events,  and  to  support  applications  which  require  microsecond 
level  response  time. 

3.2  Callouts 

Device  drivers  and  protocol  code  are  inherently  interrupt  driven,  and  most  operating  system  imple- 
mentations naturally  reflect  this  by  performing  most  of  their  work  at  software  interrupt  level.  Such 


a  solution,  while  fine  for  a  lime-sharing  system,  can  be  disastrous  in  a  real-time  system.  This  is  be- 
cause the  interrupt  handler  always  preempts  a  process  running  at  the  processor  s  base  priority,  and 
if  the  handler  does  not  happen  to  be  performing  the  highest  priority  work,  the  "priority  inversion" 
may  cause  the  more  important  processes  to  miss  their  deadlines. 

To  get  around  this  problem,  work  requested  at  interrupt  level  must  be  queued  for  later  execution 
at  the  base  processor  priority.  In  SAGE  this  is  done  through  a  callout,  which  consists  of  (pointers  to) 
the  subroutine  to  be  executed  and  its  data  arguments.  Each  callout  also  has  an  associated  priority, 
which  is  the  priority  the  subroutine  should  be  executed  at  later  on. 

Callouts  are  executed  by  a  pool  of  server  processes.  A  running  server  removes  the  highest 
priority  callout  from  the  queue,  sets  its  own  priority  to  the  priority  of  the  callout,  and  executes  the 
callout's  subroutine  to  completion.  If  a  higher  priority  callout  is  queued  before  the  currently  running 
server  has  completed  the  subroutine,  another  server  process  will  be  made  runnable  and  preempt  the 
currently  running  server. 

Because  a  callout  has  effectively  no  state  before  it  begins  running  or  after  it  has  finished,  the 
system  can  perform  several  optimizations.  For  instance,  when  context  switching  to  a  newly  readied 
server  process  that  is  about  to  execute  a  call^'il,  we  do  not  need  to  restore  most  of  that  process's 
volatile  state.  In  particular,  we  do  not  need  not  restore  any  registers  for  the  server  process  except 
for  the  program  counter  and  stack  pointer.  Likewise,  when  a  server  is  finished  executing  a  callout 
and  the  CPU  needs  to  be  rescheduled,  none  of  the  process  registers  have  to  be  saved. 

Callouts  provide  a  way  of  scheduling  light  weight  preemptible  tasks  inside  the  kernel,  without 
having  to  dedicate  a  kernel  resident  process  to  each  task. 

3.3  Scheduler 

The  SAGE  scheduler  is  preemptive,  and  it  will  always  run  the  process  that  has  the  highest  priority. 
Equal  priority  processes  are  run  in  FIFO  order,  based  on  when  the  process  was  placed  on  the  ready    , 
queue. 

Process  priorities  are  specified  by  64  bit  integer  values;  the  system  merely  compares  priority 
according  to  macros  specified  at  system  compile  time.  Using  this  mechanism,  the  system  can  support 
a  priority  based  scheduler,  a  deadline  based  scheduler,  or  for  that  matter,  any  other  policy  that  allows 
priorities  to  be  specified  in  64  bit  integers. 

By  default,  SAGE  processes  execute  at  the  processor's  base  hardware  priority,  so  all  interrupt 
handlers  effectively  have  a  higher  priority  than  any  process.  However,  a  SAGE  process  can  elevate 
its  processor  priority  as  needed  to  mask  out  interrupts. 

SAGE  also  adjusts  process  priorities  dynamically  to  avoid  priority  inversion  effects.  For  example, 
if  a  low  priority  process  is  executing  inside  a  critical  section,  and  a  high  priority  process  attempts  to 
enter  the  critical  section,  the  low  priority  process  will  have  its  priority  temporarily  raised  to  that  of 
the  high  priority  process  until  it  has  left  the  critical  section,  at  which  point  the  original  priority  will 
be  restored.  Simulation  studies  show  that  such  a  priority  inheritance  protocol  results  in  a  higher 
percentage  of  randomly  generated  tzisk  sets  meeting  their  real-time  deadlines  [11]. 

SAGE  also  allows  a  process  to  change  the  priorities  of  other  processes  through  system  calls. 
Thus  other  scheduling  policies,  such  as  round-robin  preemption,  can  be  simulated  somewhat  by  aji 
intelligent  user-level  process. 

3.4  Memory  Management 

Many  real-time  systems  avoid  using  memory  management  features  because  of  the  (supposedly)  high 
overhead.  SAGE,  however,  views  memory  management  as  an  essential  component  of  a  supervisory 
control  system,  since  it  provides  both  protection  and  fault  containment  in  a  multitasking  envi- 
ronment. In  addition,  SAGE  also  exploits  memory  management  hardware  to  improve  the  overall 
performance  and  functionality  of  the  system. 

The  major  goal  of  SAGE's  memory  management  facility  is  to  make  shared  memory  convenient 
and  easy  to  use.  Shared  memory  is  considered  essential  for  real-time  applications  because  of  the  high 
bandwidth  it  provides  for  interprocess  communication    In  addition,  shared  memory  is  important  for 
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Figure  2:  Object  naming 


robotics  supervisory  applications  because  most  servo  systems  are  based  around  shared  memory  and 
mailbox  interrupts  [2,5,1]. 

SAGE  memory  management  operations  center  around  the  notion  of  a  segment,  which  is  simply 
a  range  of  virtual  addresses.  Segments  are  named  objects,  and  they  can  be  mapped  into  a  process's 
address  space  through  an  open  system  call.  Shared  memory  is  provided  by  having  several  processes 
open  the  same  segment  simultaneously. 

Although  a  segment's  virtual  addresses  are  generally  mapped  into  a  set  of  physical  pages,  SAGE 
also  provides  control  operations  that  allow  a  segment's  addresses  to  be  mapped  into  either  bus 
memory  or  I/O  space.  Another  control  operation  allows  a  segment's  data  to  be  mapped  out  onto 
the  bus,  where  it  can  be  accessed  by  DMA  bus  masters.  Thus  the  system  provides  a  general  memory 
mapped  I/O  capability.  Furthermore,  since  users  can  define  their  own  interrupt  handlers,  it  is  very 
easy  for  applications  to  supply  their  own  device  drivers. 

Another  control  operation  allows  the  process  to  set  segment  protections  such  as  read-only,  read- 
write,  etc.  The  typical  process  will  create  at  least  three  segments  for  itself:  a  read-only  text  segment, 
a  read-write  data  segment,  and  a  read-write  stack  segment.  Segment  protection,  combined  with  the 
ability  to  invalidate  individual  pages,  allows  a  SAGE  process  to  set  firewalls  in  its  address  space. 

The  system  insures  that  a  process  (running  in  user  mode)  can  only  access  those  segments  it 
has  explicitly  requested.  This  is  done  by  allocating  a  separate  MMU  context  for  each  process,  and 
validating  only  those  addresses  that  refer  to  mapped-in  segments.  All  segments,  however,  are  mapped 
into  the  kernel's  address  space,  to  avoid  extra  overhead  in  the  kernel.  (In  the  current  implementation, 
segments  reside  at  the  same  virtual  address  in  both  the  kernel  and  the  process  address  spaces.  This 
allows  the  system  to  also  avoid  translating  process  addresses  to  kernel  addresses.  To  allow  aliasing  of 
virtual  addresses,  a  private  page  of  data  is  allocated  to  each  process  and  is  mapped  into  the  process's 
address  space  at  a  fixed  location.) 


3.5      Object  Naming 

Segments  are  just  one  type  of  object  supported  by  SAGE.  Other  SAGE  objects  include  processes, 
communication  ports,  and  synchronization  primitives. 

Objects  are  named  by  32-bit  global  names  that  behave  like  a  capability  to  the  object.  The 
naming  scheme  is  illustrated  in  Figure  2.  The  low  order  16  bits  of  the  name  contain  a  sequence 
number,  while  the  high  order  16  bits  are  an  index  into  a  global  descriptor  table  meiintained  by  the 
kernel. 

Each  descriptor  entry  contains  a  reference  count,  a  sequence  number,  and  access  rights  to  the 
object.   The  reference  count  allows  descriptors  to  be  garbage  collected  when  a  process  exits.   The 


sequence  number  prevents  accidental  re-use  of  the  descriptor,  and  is  identical  to  the  low  order  16 
bits  of  the  descriptor  name.  The  access  rights  include  read,  write,  and  owner  permission. 

A  descriptor  also  contains  a  pointer  to  an  object  structure.  Each  object  contains  a  reference  count, 
a  pointer  to  an  object's  private  data,  and  a  pointer  to  a  function  switch  table  that  implements  the 
object's  operations.  Object  reference  counts  are  needed  in  addition  to  descriptor  reference  counts 
since  there  may  be  many  descriptors  referencing  a  single  object.  Every  time  a  descriptor  is  deleted, 
the  object  reference  count  is  decremented  When  the  reference  count  becomes  zero,  the  destroy 
routine  (in  the  object  switch  table)  is  invoked.  The  object  switch  table  also  contains  entries  for 
generic  operations  such  as  read  and  write,  as  well  as  a  control  operation  which  provides  object 
specific  functions. 

A  process  only  needs  to  know  the  32-bit  object  name  in  order  to  perform  an  operation  on  an 
object.  These  can  be  passed  in  shared  memory  or  returned  by  a  name-server.  During  the  course  of 
an  operation,  the  reference  count  on  the  descriptor  is  temporarily  incremented.  This  prevents  an 
object  from  being  deleted  out  from  underneath  the  invoker. 

The  SAGE  global  naming  scheme  is  especially  convenient  for  shared  memory  applications.  Typ- 
ically, an  object  need  only  be  initialized  once,  and  then  any  process  can  access  the  object  by  just 
using  the  object  name.  In  contrast,  many  systems  reference  objects  by  process-specific  small  integer- 
valued  descriptors.  However,  global  names  are  still  needed  in  such  systems,  since  the  application 
has  to  obtain  a  process-dependent  descriptor  before  the  object  can  be  referenced.  This  extra  initial- 
ization not  only  leads  to  greater  overhead,  but  also  complicates  the  programming  of  shared  routines 
that  manipulate  global  objects. 

When  process-dependent  naming  of  objects  is  required,  virtual  memory  can  be  used  to  achieve 
the  same  effect  by  arranging  for  the  same  variable  location  to  be  mapped  to  different  physical 
addresses  in  different  processes. 

3.6      Synchronization  Primitives 

SAGE  provides  two  b2isic  facilities  for  process  synchronization:  a  locking  facility,  used  for  atomic 
critical  sections,  and  a  monitor-like  facility,  whereby  processes  can  suspend  and  resume  themselves 
while  waiting  for  a  particular  condition  to  become  true.  The  facilities  are  intended  to  avoid  priority 
inversion  effects  and  to  allow  application  dependent  synchronization  primitives  to  be  built  with  a 
minimum  of  system  overhead.  They  are  similar  to  those  proposed  in  [12]. 

3.6.1      Locking  Primitives 

The  locking  primitives  are  used  to  insure  that  critical  sections  are  atomically  executed.  In  order  to 
avoid  system  call  overhead,  the  locking  primitives  are  designed  as  two-part  operations  In  the  lock 
operation,  the  process  first  attempts  to  lock  the  critical  section  directly,  and  only  if  that  operation 
fails,  will  the  process  trap  to  the  kernel.  Likewise,  the  unlock  operation  will  only  perform  a  system 
call  if  other  processes  are  waiting  to  enter  the  critical  section. 

Before  using  the  locking  primitives,  a  process  creates  a  shared  lock  structure  (named  1  below). 
Then  to  lock  a  critical  section,  the  process  performs  a  68020  compare-and-swap  operation: 

while    (cmpandsffpCl .mutex,   NULL,    sell)    !=   HULL) 

kernel-bloc)c(tl)  ;  /»   sets   1.  waiters   */ 

Here  l.mutex  contains  the  descriptor  name  of  the  process  holding  the  critical  section,  or  zero  if  the 
critical  section  is  unoccupied.  If  the  comparison  against  zero  succeeds,  the  variable  is  atomically 
updated  with  the  process's  identifier  eell.  Otherwise,  a  kernel  routine  is  called  (with  the  address  of 
the  lock  structure)  to  suspend  the  process.  When  the  process  is  resumed  the  cmpandssap  is  retried, 
since  a  software  interrupt  may  awaken  the  process  prematurely. 
The  process  unlock  operation  is: 

l.mutex   =  lULL; 

±i    (1. waiters  ==  TRUE) 

kernel-resume (tl) ; 


This  clears  l.mutex  with  an  atomic  write  operation,  and  if  there  are  other  processes  waiting  to  enter 
the  critical  section,  calls  the  kernel  to  wake  up  the  highest  priority  process. 

Note  that  there  is  a  potential  race  condition  here.  Between  the  time  the  cmpandswap  test  fails 
and  kernel-block  is  called,  the  process  already  inside  the  critical  section  could  leave  the  critical 
section  and  find  the  queue  empty  (1. waiters  ==  TRUE).  In  this  case  kernel-resume  would  not  be 
called,  and  the  process  suspending  itself  may  never  be  awakened.  Therefore,  kernel-block  must 
re-test  whether  the  critical  section  is  available  as  follows: 

spin-lockO ;  /♦   inhibits  preemption  */ 

1. waiters   =  TRUE; 

holder   =   cmpandsopCl .mutex,   NULL,    self); 

if    (holder    !=  NULL)    i 

if    (self.pri  >  holder. pri) 

setpriority (holder,  self.pri); 

enqueue(self ,  l.waitq);   /*  put  to  sleep  */ 

reschedule-and-spin-unlock() ; 
>  else  i 

1. waiters  =  q-not-enipty(l.waitq)  ; 

l.mutex  =  NULL; 

spin-unlockO ;      /*  ends  busy-wait  ♦/ 
> 

If  the  critical  section  is  still  occupied,  the  priority  of  the  process  holding  the  critical  section  is  raised 
as  necessary  to  avoid  priority  inversion  effects.  If  the  critical  section  is  now  free,  however,  we  clear 
the  variable  and  retry  the  whole  test. 

kernel-resume  will  restore  the  process's  original  priority,  and  will  wake  up  the  highest  priority 
process  waiting  for  this  critical  section: 

spin-lockO ; 

re8tore-priority(Belf ) ; 
next  =  dequeue (l.waitq) 
if    (next) 

resume-process (next) ; 
1. waiters  =   q-not-empty (1 .waitq) ; 
spin-unlockO  ; 

Note  both  kernel  routines  are  non-preemptible  and  execute  within  a  busy-wait  lock  to  avoid  potential 
race  conditions. 

In  the  usual  case,  no  other  processes  are  waiting  for  the  critical  section,  and  only  ten  68020 
instructions  are  required  to  implement  the  lock-unlock  pair.  In  particular,  system  calls  are  avoided. 
This  is  important  if  shared  memory  is  to  be  used  effectively  between  tightly  coupled  processes,  which 
is  typically  the  case  in  robotics  applications. 

Although  not  shown  here,  it  is  also  possible  to  implement  the  locking  primitives  in  a  similar 
fashion  using  a  test-and-set  operation  instead  of  compare-and-swap.  Therefore  this  technique  can 
be  applied  in  a  reasonably  processor-independent  fashion. 

3.6.2      Condition  Variables 

The  other  synchronization  primitive  is  the  condition  variable.  It  is  intended  to  provide  monitor- 
like synchronization  capabilities  to  language  run-time  systems.  In  particular,  there  are  routines 
to  suspend  a  process  waiting  for  a  condition  to  become  true,  wake  up  a  single  process  awaiting 
that  condition,  or  wake  up  all  processes  waiting  for  that  condition.  The  implementation  of  condition 
variables  is  again  a  two-part  design  that  distributes  the  code  and  data  structures  between  the  process 
emd  the  kernel. 

Before  using  a  condition  variable,  a  process  must  create  the  shared  structure  (named  c  be- 
low). Then  to  suspend  a  process  awaiting  a  condition,  a  process  (inside  an  atomic  section  1)  sets 


c. Baiters,  indicating  that  processes  are  waiting  on  the  critical  section,  and  notes  the  current  value 
of  the  condition's  event  count: 

value   =   c. event; 
c. waiters   =   TRUE; 
uiilock(ftl)  ; 
kernel-wait (tc,   value); 

The  process  then  calls  the  kernel,  passing  as  arguments  the  name  of  the  condition  and  the  event 
count  value.  To  wake  up  a  process  awaiting  a  condition,  the  process  will  check  the  c. waiters  flag 
and  call  the  kernel  if  there  are  any  processes  awaiting  that  condition: 

if    (c.waiters  ==   TRUE) 

kernel-signal(Ac) ; 

kernel-signal  then  resumes  the  highest  priority  process  from  the  condition  wait  queue,  and  incre- 
ments the  event  count; 

spin-lockO  ; 

next  =  dequeueCc . waitq) ; 

ii  (next) 

resune-process(next) ; 
c.waiters  =  q-not-empty(c. waitq) ; 
c . event++ ; 
spin-unlock () ; 

The  system  call  to  awake  all  processes  awaiting  a  condition  works  similarly,  but  instead  of  awaking 
just  the  highest  priority  process,  readies  all  processes  on  the  condition  wait  queue. 

The  event  count  is  used  to  avoid  a  waiting-wake  up  race  condition  between  the  wait  and  signal 
operations,  kernel-wait  will  suspend  a  process  only  if  the  event  count  argument  is  the  same  as  the 
current  value  of  the  event  count: 

spin-lockO  ; 

c.waiters  =  TRUE; 

ii    (value  ==  c. event)  i 

enqueue(self ,  c. waitq); 

reschedule-smd-spin-uulockO ; 
}   else  -i 

c.waiters  =  q-not-empty (c. waitq) ; 

c. event++ ; 

spin-unlockO  ; 
> 

Therefore  if  kernel-signal  is  called  after  the  condition  is  tested  but  before  the  process  has  put 
itself  to  sleep,  the  wake  up  will  not  be  missed  because  the  event  count  has  changed. 

Note  that  although  there  is  no  waiting-wake  up  race,  there  are  some  cases  where  more  than  one 
process  can  be  awakened.  Therefore,  a  process  returning  from  kernel-wait  must  re-test  the  condi- 
tion. By  not  insisting  on  the  more  restrictive  semamtics,  we  obtain  a  very  efficient  implementation. 
In  addition,  by  not  disabling  interrupts,  an  arbitrarily  long  time  can  elapse  between  the  time  the 
critical  section  is  exited  and  the  time  the  process  is  suspended.  In  particular,  other  conditions  and 
critical  sections  can  be  tested.  Because  SAGE  provides  a  system  call  that  can  wait  for  a  number 
of  conditions  to  become  true  (or  critical  sections  to  become  free),  a  process  can  effectively  manage 
several  conditions  at  once. 

Many  flexible  mechanisms  can  be  built  on  top  of  these  synchronization  primitives,  including  a 
bounded-buffer  style  message-passing  interface. 


3.7  Timing  Facilities 

SAGE  provides  both  absolute  and  relative  timer  facilities.  An  absolute  timer  expires  at  a  given 
time,  while  a  relative  timer  expires  after  a  specified  interval  of  time.  Processes  can  synchronously 
block  until  the  timer  expires,  or  request  asynchronous  notification  of  the  timer  expiration  through 
a  software  interrupt. 

When  setting  a  relative  timer,  the  current  time  is  returned  atomically,  allowing  a  process  to  know 
exactly  when  the  timer  will  expire.  This,  coupled  with  the  ability  to  set  absolute  timers,  allows  a 
SAGE  application  to  generate  precise  interval  timers  without  incurring  any  clock  drift. 

All  time  arguments  are  specified  to  microsecond  resolution,  since  high  resolution  timers  are 
needed  to  avoid  quantization  errors.  Absolute  time  values  are  specified  in  terms  of  the  number  of 
microseconds  elapsed  since  the  system  has  booted.  To  convert  to  local  time,  the  kernel  maintams  a 
global  time  ofi'set,  which  when  added  to  the  absolute  time,  gives  the  current  Greenwich  Mean  Time. 

To  achieve  high  resolution  on  the  alarm  and  time  of  day  functions,  SAGE  uses  two  hardware 
timers,  with  one  timer  providing  time  of  day  information,  and  the  other  timer  providing  an  event 
timer  interrupt.  Conceptually  then,  the  time  of  day  function  is  provided  by  simply  reading  the  time 
of  day  register.  In  practice,  however,  the  register  is  a  counter  clocked  at  a  fast  frequency,  and  so 
can  wrap  around  quickly.  Thus  we  must  also  maintain  an  additional  counter  in  kernel  memory  to 
record  the  overflow. 

Internally,  all  alarm  events  are  stored  on  a  queue,  sorted  in  increasing  absolute  time.  The  interval 
timer  is  then  programmed  to  interrupt  at  the  time  given  by  the  first  queue  entry.  The  hardware 
clock  handler  removes  all  expired  entries  from  the  queue  (as  determined  by  a  simple  comparison), 
and  reprograms  the  interval  timer  to  interrupt  at  the  next  timer  event  given  by  the  new  entry  at 
the  head  of  the  wait  queue.  In  the  current  implementation,  the  effective  resolution  of  the  time  of 
day  and  interrupt  timers  is  limited  by  system  call  overhead  to  roughly  50  microseconds. 

In  addition  to  providing  wall  clock  timers,  SAGE  also  provides  several  profiling  and  instrumen- 
tation timers.  However,  because  the  number  of  hardware  timers  is  limited,  these  profiling  timers  are 
performed  by  a  sampling  routine  called  at  a  configuration-dependent  frequency,  which  is  usually  set 
at  100  HZ.  Therefore,  the  profiling  timers  have  relatively  coarse  resolution. 

3.8  Distributed  Communications 

Supervisory  control  systems  work  in  an  inherently  distributed  environment,  so  it  is  important  that 
the  system  supports  network  communications.  SAGE  was  designed  to  efficiently  support  multiple 
protocols  running  on  different  physical  networks. 

Multiple  protocols  need  to  be  supported  for  several  reasons: 

•  No  single  protocol  seems  appropriate  for  all  real-time  applications,  because  of  the  inherent 
tradeoff  between  the  reliability  and  throughput  of  the  protocol. 

•  Certain  hosts  have  not  implemented  certain  protocols.  For  an  application  to  communicate 
with  these  hosts,  it  is  generally  easier  to  support  the  host's  protocol  in  SAGE  than  to  modify 
the  host's  software. 

Currently,  several  of  the  standard  Internet  protocols  have  been  implemented,  including  IP,  ICMP, 
and  UDP.  In  addition,  a  reliable  datagram  protocol  (built  on  top  of  UDP)  has  been  implemented 
for  remote  procedure  calls.  SUN  RPC  and  TCP  are  currently  being  ported  to  the  system. 

For  all  the  protocols,  the  kernel  takes  care  of  low-level  network  details  such  as  routing,  checksums, 
fragmentation  and  assembly,  etc.  The  user  is  presented  with  a  simple  open-close-read-write-control 
interface  to  the  network.  In  addition,  priorities  can  be  etssociated  with  the  communication  ports. 
The  protocol  processing  for  a  port's  incoming  packets  then  takes  place  at  the  specified  priority. 

The  system  also  provides  a  number  of  kernel-level  utility  routines  for  supporting  protocols.  In 
most  cases,  data  inside  the  kernel  is  passed  by  reference  rather  than  copying,  and  routines  exist 
to  efficiently  copy,  split,  trim  and  assemble  packets.  Generic  interfaces  to  device  drivers  are  also 
provided. 


3.9  Process  Control  Primitives 

Process  control  primitives  are  used  for  process  creation,  debugging,  handling  exceptions,  and  chang- 
ing process  priority.  These  primitives  are  invoked  through  a  standard  control  operation  on  a  process 
descriptor. 

Initially,  a  process  is  created  in  a  stopped  and  traced  state.  The  parent  process  (or  another 
process  with  access  to  the  object)  can  then  set  up  the  initial  state  of  the  object.  Typically,  this 
involves  copying  descriptors,  instantiating  exception  handlers,  and  initializing  the  process's  address 
space  and  registers. 

A  process  can  also  be  placed  in  a  stopped  and  traced  state  after  a  process  has  been  started. 
Commands  allow  a  user-level  debugger  to  single-step,  set  breakpoints,  and  peak  and  poke  a  process's 
address  space.  More  than  one  process  can  be  traced  simultaneously. 

Exceptions  are  handled  through  a  software  interrupt  mechanism  patterned  after  the  Berkeley 
UNIX  signal  facilities.  When  a  signal  is  delivered  to  a  process,  the  system  arranges  for  the  process 
to  jump  to  a  specified  trap  handler.  A  signal  can  be  raised  by  a  process  with  appropriate  access 
rights,  or  generated  by  the  system  in  response  to  a  hardware  fault  such  as  a  bad  memory  reference, 
division  by  zero,  etc.  If  the  process  has  not  set  up  a  signal  handler,  the  process  will  be  terminated. 

SAGE  adopted  the  Berkeley  UNIX  model  because  it  is  a  familiar  one  to  most  programmers. 
However,  the  SAGE  interface  differs  from  the  UNIX  interface  in  two  significant  ways.  First,  SAGE 
allows  other  processes  to  examine  and  modify  the  signal  state  of  a  particular  process,  while  in  UNIX, 
this  can  only  be  done  by  the  process  itself.  This  turns  out  to  be  useful  for  handcrafting  new  processes 
and  controlling  multiple  threads  running  in  the  same  address  space.  Secondly,  SAGE  allows  a  process 
to  atomically  change  its  priority  upon  receiving  a  signal.  This  allows  more  deterministic  scheduling 
in  response  to  signal  delivery. 

3.10  Interprocess  Communication 

In  addition  to  shared  memory  and  distributed  communications,  the  system  provides  a  procedure 
call  message  discipline  for  interprocess  communications.  A  server  wishing  to  provide  a  facility  will 
create  a  procedure  call  object.  The  object  serves  as  a  rendezvous  point  for  clients  and  a  server  to 
communicate. 

A  client  sends  a  message  to  a  server  by  performing  a  system  call.  Conceptually,  this  operation 
places  the  client  process  on  a  queue  of  processes  waiting  to  be  serviced.  When  the  server  is  ready 
to  accept  a  message,  the  client's  message  is  transferred  between  the  two  processes  When  the  server 
has  finished  processing  the  message,  it  returns  a  reply,  which  is  then  copied  back  to  the  client.  At 
this  point,  the  client  returns  from  the  system  call  and  resumes  operation. 

The  kernel  appends  an  unforgeable  header  to  the  client's  request;  this  allows  one  server  (or  a 
multi-threaded  server)  to  handle  a  number  of  different  streams  simultaneously.  The  system  is  careful 
to  queue  procedure  call  requests  in  priority  order,  so  that  these  requests  are  serviced  in  the  proper 
order.  In  addition,  the  server  inherits  the  priority  of  the  client  for  the  duration  of  the  procedure 
call,  in  order  to  mitigate  priority  inversion  effects. 

Procedure  call  objects  allow  system  facilities  to  be  implemented  outside  the  kernel  in  a  real-time 
fashion.  Although  it  is  possible  to  emulate  procedure  call  objects  using  the  shared  memory,  syn- 
chronization, and  process  control  primitives,  client-server  applications  are  so  common  that  the  extra 
efficiency  gained  by  implementing  procedure  call  objects  in  the  kernel  seems  worthwhile.  In  addi- 
tion, the  remote  procedure  call  paradigm  generalizes  easily  to  a  distributed  environment,  possibly 
allowing  SAGE  to  interact  with  other  real-time  systems  in  the  future. 

3.11  Device  Support 

Currently,  both  VME  and  Multibus  interfaces  are  supported.  Device  drivers  can  be  implemented 
using  either  kernel  processes,  user  processes,  interrupt  handlers,  or  kernel  callouts  (or  a  combination 
of  all  these  facilities).  The  memory  management  primitives  also  provide  support  for  DMA  devices, 
which  need  to  access  pages  from  the  bus. 
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The  standard  system  also  provides  support  for  several  serial  line  disciplines,  including  the  4.3  BSD 
tty  interface  (for  interactive  use).  In  addition,  the  system  supports  a  block  line  discipline  for  high 
bandwidth  applications.  Using  this  interface,  a  read  request  will  not  return  until  either  the  specified 
number  of  characters  or  a  character  in  a  terminator  mask  has  been  read.  This  discipline  allows 
the  system  to  efficiently  handle  a  number  of  serial-line  protocols,  since  needless  context  switching 
between  the  kernel  and  the  user  process  can  be  avoided. 

4      Support  Software 

The  SAGE  support  software  encompasses  all  code  that  is  not  kernel-resident  in  the  satellite  processor 
board.  It  can  roughly  be  broken  down  into  three  major  components: 

C  Run-Time  Support  These  modules  implement  standard  facilities  such  as  file  I/O,  floating  point 
math  libraries,  heap  allocation,  etc.  that  are  assumed  to  be  available  in  UNIX  C  environments. 
These  routines  can  either  be  explicitly  called  by  the  programmer,  as  in  the  case  of  the  standard 
I/O  library,  or  implicitly  called  by  the  compiler,  as  in  the  case  of  floating  point  support. 

System  Support  These  libraries  and  programs  implement  a  wide  range  of  services  that  can  be 
used  by  SAGE  applications.  Typical  services  are  remote  filesystem  access,  process  loading, 
and  kernel  bootstrapping. 

Program  Development  Utilities  These  programs  facilitate  writing  and  debugging  of  SAGE  ap- 
plications. Some  of  these  programs,  such  as  the  cross-compiler,  run  on  the  host,  while  other 
programs,  such  as  the  debugger,  run  on  the  SAGE  satellite. 

In  this  section  we  describe  some  of  the  more  noteworthy  features  of  this  code. 

4.1  UNIX  Emulation 

The  SAGE  C  library  emulates  a  number  of  Berkeley  UNIX  system  calls,  including  the  filesystem- 
related  calls  (open,  close,  read,  write,  unlink,  etc.)  as  well  as  the  signal  handling  calls.  The  filesystem 
calls  are  implemented  through  remote  procedure  calls  made  to  a  SUN  fileserver  process,  while  the 
signal  handling  code  is  built  on  top  of  the  process  control  primitives. 

Providing  a  primitive  UNIX  emulation  has  proved  to  be  very  convenient:  researchers  can  use 
SAGE  quickly,  without  having  to  learn  a  completely  new  operating  system. 

4.2  Floating  Point 

The  SAGE  math  library  is  really  just  a  modification  of  the  SUN-3  math  library.  Since  most  of  the 
code  in  the  library  is  operating-system  independent,  we  only  needed  to  modify  the  few  routines  that 
interfaced  to  the  system  (e.g.,  to  determine  which  floating  point  co-processor  was  attached  to  the 
system). 

The  advantage  of  adapting  the  SUN-3  routines  is  that  it  allows  us  to  use  any  of  the  SUN  compilers 
without  any  modifications.  Currently,  SAGE  applications  are  written  in  C,  Fortran,  and  C-I-+,  and 
support  either  the  68881  floating  point  chip  or  software  floating  point. 

4.3  Process  Loader 

When  the  system  is  initially  loaded,  the  first  process  to  run  is  a  process  loader,  which  continuously 
listens  for  load  requests  from  other  hosts  in  the  Ethernet.  When  a  request  is  received,  a  new  process 
is  created,  the  code  downloaded  via  a  reliable  protocol  to  the  SAGE  system,  and  the  process  begins 
executing  in  the  new  context. 

The  process  loader  executes  asynchronously  of  other  processes,  so  new  processes  can  be  dy- 
namically downloaded  at  any  time.  This  was  considered  an  important  capability  for  a  supervisory 
environment,  which  must  often  react  to  external  events. 
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SAGE 

SUN  3/160 

Null  system  call  (usees) 

49 

121 

Null  process  (msecs) 

1 

10 

UDP  loopback  (msecs/msg) 
4  bytes 
4K  bvtes 

1.84 
15.92 

1.95 
8.22 

Process  switch  (usees) 
user  to  user 
queue/start  callout 

170 
170 

178+call 

Message  passing  (usecs/msg) 
4  bytes  qsize  1 
4  bytes  qsize  20 

920 

90 

— 

Table  1:  SAGE  Benchmarks 

4.4      Debugging  Tools 

The  system  supports  a  number  of  debugging  tools.  A  kernel  resident  debugger  can  be  loaded  with 
the  system  to  allow  debugging  of  both  kernel  and  user  processes.  The  debugger,  based  on  ddl, 
provides  an  assembly  language  debugger  which  can  disassemble  instructions,  print  data  in  several 
formats,  and  breakpoint  or  single  step  code.  Multiple  symbol  tables  (corresponding  to  a  number  of 
processes)  can  be  handled  simultaneously. 

In  addition,  user-level  debuggers  can  be  constructed  on  top  of  the  system's  process  control 
primitives.  The  system  provides  facilities  to  peek  and  poke  a  process's  address  space,  trap  exceptions, 
set  breakpoints,  etc.  A  facility  exists  to  remotely  trace  processes  from  the  UNIX  development  host. 

The  system  also  provides  several  profiling  tools,  including  UNIX  prof  and  gprof.  This  is  useful 
for  performance  tuning;  however,  because  the  interface  is  not  completely  unobtrusive,  task  sets  with 
very  tight  timing  constraints  may  fail  when  profiling  is  enabled. 

Finally,  the  kernel  contains  a  number  of  instrumentation  counters  which  can  be  examined  with 
the  system  debugger.  In  addition,  trace  buffers  and  debugging  print  outs  allow  parts  of  the  system 
to  analyzed  while  the  kernel  is  active. 

5      Current  Status 

A  single  processor  implementation  of  SAGE  is  now  running  on  several  different  Pacific  Microsystems 
68020-ba8ed  processor  boards,  including  both  Multibus  and  VMEbus  systems.  The  hardware  com- 
ponents used  by  SAGE  include  the  memory  management  unit  (for  protection  and  memory  mapped 
I/O),  dual  ported  memory,  two  high  resolution  timers,  and  on-board  software  interrupts.  These 
components  are  found  in  most  minicomputers  and  many  microcomputers. 

5.1      Benchmarks 

Preliminary  benchmarks  are  illustrated  in  Table  1.  As  a  comparison  yardstick,  we  also  (where 
applicable)  give  the  results  of  similar  benchmarks  run  on  a  SUN  3/160  under  the  SUN  3.4  operating 
system.  Both  systems  use  essentiadly  the  same  processor;  a  68020  clocked  at  16.7  MHz  with  no  wait- 
state  on-board  memory.  Of  course,  comparisons  between  systems  should  not  be  taken  too  seriously, 
since  the  systems  were  designed  to  meet  totally  different  goals. 

Two  benchmarks  are  of  particular  interest.  First,  note  that  over  5000  context  switches  per-second 
can  be  executed  between  user- processes  including  system  call  overhead.  This  implies  that  the  system 
should  be  more  than  able  to  handle  events  occurring  at  the  millisecond  level,  and  in  particular,  the 
system  should  not  thrash  with  excessive  context  switch  overhead.  In  addition,  the  callout  time  (the 
time  to  queue  a  callout,  context  switch,  and  execute  the  first  instruction  performing  useful  work)  is 
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roughly  the  same  as  a  normal  context  switch  time.  In  other  words,  the  callout  optimizations  make 
lightweight  task  creation  no  more  expensive  than  a  normal  context  switch. 

Another  number  of  interest  is  the  message  passing  time,  which  gives  the  time  required  for  a 
long  word  message  to  be  sent  between  processes.  The  particular  message  passmg  scheme  used  is 
a  classical  bounded-buffer  implementation  on  top  of  shared  memory  using  the  SAGE  locking  and 
condition  synchronization  primitives.  Note  than  for  a  queue  size  of  one,  two  context  switches  per 
message  are  incurred,  because  both  the  empty  and  full  states  of  the  queue  are  signaled.  Nevertheless, 
the  message  time  is  still  quite  respectable  compared  to  other  systems.  However,  as  the  queue  size 
becomes  larger  and  the  messages  are  buffered,  less  context  switches  are  incurred,  and  the  cost  of 
context  switching  is  amortized  over  all  the  messages.  Therefore  in  many  cases,  SAGE  processes  can 
use  a  message  passing  discipline  that  is  no  more  costly  than  shared  memory. 

Although  benchmarks  are  interesting,  they  can  not  measure  perhaps  the  most  important  aspect 
of  the  system:  reliability  in  meeting  real-time  deadlines.  Indeed,  SAGE's  only  performance  goal  has 
been  to  support  the  type  of  supervisory  control  applications  performed  in  our  laboratory.  In  this 
regard,  SAGE  has  been  successful.  In  particular,  one  68020-based  system  simultaneously  handles 
several  thousand  interrupts  a  second  and  a  number  of  active  network  connections,  all  while  still 
providing  real-time  response  on  the  order  of  milliseconds. 

5.2      Applications 

SAGE  is  currently  being  used  in  several  projects  at  the  NYU  Robotics  laboratory,  including  con- 
trolling a  PUMA  560  robot  and  an  experimental  four-finger  manipulator. 

The  PUMA  robot  arm  is  controlled  through  the  VAL-II  alter  and  supervisory  control  serial-line 
interfaces.  In  addition,  the  SAGE  system  integrates  a  number  of  sensors,  including  a  force-sensing 
wrist,  an  image  processor,  and  the  graphics  capabilities  of  the  SUN  workstation. 

In  the  four-finger  manipulator,  SAGE  directly  controls  several  stepper  motors  with  multibus 
interfaces  by  mapping  the  devices  directly  into  the  user  address  space.  In  addition,  force-sensing 
capabilities  are  provided  through  conditioned  strain  gauges  that  are  read  with  A-D  converters. 

Both  applications  are  inherently  multi-tasked  and  make  use  of  SAGE  memory  management 
facilities,  interface  capabilities,  real-time  scheduling  facilities,  and  system  support  software. 

6      Conclusion 

It  is  reasonable  to  ask  whether  a  classical  operating  system  design  even  makes  sense  in  the  context 
of  robotics  supervisory  control.  Experience  with  SAGE  and  the  applications  built  in  our  lab  seems 
to  indicate  that  such  a  design  is  useful.  Indeed,  most  of  the  time  spent  in  building  supervisory 
applications  seems  to  be  concerned  with  interfacing  with  other  machines  and  devices.  SAGE  speeds 
up  the  process  of  interfacing  by  providing  generic  device  and  system  routines. 

However,  even  in  SAGE,  where  the  system  provides  a  great  deal  of  low  level  device  support, 
most  of  the  "application"  code  running  on  the  system  is  still  concerned  with  the  vagaries  of  device 
interfaces.  This  situation  will  persist  until  standards  are  adapted  throughout  the  industry. 

Some  of  the  more  novel  features  of  SAGE — memory  meinagement,  uniform  object  naming,  user- 
cissisted  synchronization  variables,  and  real-time  network  communications — proved  to  be  quite  useful 
in  our  applications.  In  the  past,  many  of  these  features  have  not  been  included  in  existing  systems 
on  the  grounds  of  performance  or  cost.  However,  the  declining  cost  of  hardware  makes  such  features 
practical,  and  hopefully  future  systems  that  address  supervisory  control  applications  will  contain 
such  features. 

SAGE  is  still  an  evolving  system,  and  in  the  future  we  hope  to  port  SAGE  to  several  other 
types  of  processor  boards  and  MMU  types.  In  addition,  we  would  like  to  extend  SAGE  to  a  tightly 
coupled  multiprocessor  system  on  a  VME  bus.  This  should  be  a  manageable  project,  since  the 
kernel  is  already  preemptible  in  most  places,  and  therefore  is  already  coded  in  a  style  similar  to  that 
needed  by  a  shared-memory  multiprocessor  systems  (e.g.,  locks  are  used  when  accessing  shared  data 
structures). 
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We  would  also  like  to  extend  the  kernel  in  several  ways.  The  memory  management  facilities 
could  benefit  from  some  of  the  copy-on-wrile  techniques  used  in  systems  such  as  Accent  [13],  and 
would  allow  passing  data  between  address  spaces  in  a  safe  fashion.  We  would  also  like  to  investigate 
extending  the  IPC  model  from  a  simple  remote  procedure  call  interface  to  a  queued  message  discipline 
in  a  way  that  avoids  priority  inversion  effects. 

Finally,  some  of  the  support  tools  should  be  improved.  For  example,  the  link  editor  (Berkeley 
UNIX  Id)  only  supports  text,  data,  and  stack  segments,  which  makes  it  difficult  to  code  multi- 
threaded applications  which  need  multiple  segments.  In  addition,  the  assembly  level  language  de- 
bugger should  be  replaced  by  a  more  sophisticated  line  oriented  debugger  such  as  gdb.  Finally,  new 
debugging  and  performance  analysis  tools  need  to  be  developed.  Writing  highly  concurrent  programs 
is  a  difficult  proposition  at  best,  and  currently  existing  tools  are  barely  adequate  for  debugging  and 
reasoning  about  such  systems. 
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